On the Uniform Convergence of Consistent Confidence Measures

Many classification algorithms produce confidence measures in the form of conditional probability of labels given the features of the target instance. It is desirable to be make these confidence measures calibrated or consistent, in the sense that they correctly capture the belief of the algorithm in the label output. For instance, if the algorithm outputs a label with confidence measure p for n times, then the output label should be correct approximately np times overall. Calibrated confidence measures lead to higher interpretability by humans and computers and enable downstream analysis or processing. In this paper, we formally characterize the consistency of confidence measures and prove a PAC-style uniform convergence result for the consistency of confidence measures. We show that finite VCdimension is sufficient for guaranteeing the consistency of confidence measures produced by empirically consistent classifiers. Our result also implies that we can calibrate confidence measures produced by any existing algorithms with monotonic functions, and still get the same generalization guarantee on consistency.